Skip to content

Run Hermes adapter through local harness - #35

Open
afrog33k wants to merge 2 commits into
openclaw:mainfrom
afrog33k:main
Open

Run Hermes adapter through local harness#35
afrog33k wants to merge 2 commits into
openclaw:mainfrom
afrog33k:main

Conversation

@afrog33k

@afrog33k afrog33k commented Jul 1, 2026

Copy link
Copy Markdown

Summary

  • Wire registered non-OpenClaw adapters through the generic harness lifecycle instead of rejecting them as known-but-unwired.
  • Add Hermes local adapter configuration from environment and a local verification shim for file/execution checks.
  • Prepend explicit staged-workspace guidance to Hermes MiniSWERunner prompts so local runs inspect the task workspace first.

Validation

  • /Users/reckon/miniconda3/bin/python -m pytest tests/test_harness.py tests/test_hermes_adapter.py -q -> 21 passed
  • /Users/reckon/miniconda3/bin/python -m pytest -q -> 318 passed, 5 skipped
  • Local Hermes rerun: t2-msg-summarize-thread improved from 0.63704 to 1.000 with the workspace-guidance patch.

Notes

  • This keeps OpenClaw behavior intact and routes adapter != "openclaw" through the adapter lifecycle.
  • Memory/session/cron/browser checks still need adapter-aware verification before broad non-file task sweeps.

@afrog33k
afrog33k requested a review from a team as a code owner July 1, 2026 11:36
@clawsweeper

clawsweeper Bot commented Jul 1, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex review: needs changes before merge. Reviewed August 5, 2026, 4:36 AM ET / 08:36 UTC.

ClawSweeper review

What this changes

The PR enables registered local adapters such as Hermes to run through the benchmark harness, adds workspace-first prompt guidance, and adds file-oriented lifecycle tests.

Merge readiness

⛔ Blocked by patch quality or review findings - 5 items remain

Keep open: current main still rejects Hermes as unwired, but this branch incorrectly scores any supported adapter state query through a failing gateway shim instead of the adapter verifier.

Priority: P2
Reviewed head: 67ff264005ab6b55679299ba4e7fcd7cbb1276bf

Review scores

Measure Result What it means
Overall readiness 🦪 silver shellfish (2/6) A real local Hermes result is documented, but the unresolved P1 scoring path makes the patch not ready to merge.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): The PR body records an after-fix local Hermes benchmark result; add redacted output for the stateful regression after the scoring repair.
Patch quality 🦪 silver shellfish (2/6) 1 actionable review finding remain.

Verification

Check Result Evidence
Real behavior Verified Sufficient (live_output): The PR body records an after-fix local Hermes benchmark result; add redacted output for the stateful regression after the scoring repair.
Evidence reviewed 6 items Current main does not implement the requested execution path: Current main explicitly rejects registered adapters that are not in the executable set, including Hermes, so this PR's central capability is not already implemented.
Adapter teardown precedes local completion scoring: The PR tears down the adapter before calling completion scoring with _LocalVerificationClient, leaving no path for state assertions to invoke the active adapter verifier.
Completion scoring still uses gateway-based state checks: score_task_run passes its client to verify_completion, whose memory, session, cron, and gateway assertions call gateway methods rather than AgentAdapter.verify_state_query.
Findings 1 actionable finding [P1] Route state queries through the active adapter
Security None None.

How this fits together

ShellBench runs benchmark tasks through an agent adapter, then scores workspace artifacts and persistent state. This PR changes how non-OpenClaw adapters enter that lifecycle and how their outcomes reach completion scoring.

flowchart LR
  Tasks[Benchmark tasks] --> Harness[Benchmark harness]
  Harness --> Adapter[Selected agent adapter]
  Adapter --> Workspace[Task workspace]
  Adapter --> State[Adapter state verification]
  Workspace --> Scoring[Completion scoring]
  State --> Scoring
  Scoring --> Results[Benchmark results]
Loading

Before merge

  • Route state queries through the active adapter (P1) - The local path tears down the adapter, then scores with _LocalVerificationClient; memory, session, cron, and gateway assertions therefore bypass verify_state_query and fail for any local adapter that supports them. Keep the adapter/context available through scoring and add stateful-adapter coverage. This is the prior blocker still present on the unchanged head.
  • Resolve merge risk (P1) - Merging as written can report incorrect completion results for any local adapter that declares a state capability, because its verifier is bypassed and the gateway shim fails the assertion instead.
  • Complete next step (P2) - A narrow mechanical repair can preserve this PR's local-adapter lifecycle while correcting the unsupported state-scoring path.
  • Improve patch quality - Route state queries through the live adapter verifier and add a stateful local-adapter regression test.
  • Improve patch quality - Post redacted output from the repaired stateful run if the implementation changes the reported benchmark behavior.

Findings

  • [P1] Route state queries through the active adapter — clawbench/harness.py:644
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production vs test delta production +247/-10; tests +103/-7 Most of the 4-file change adds lifecycle code, while the added coverage currently exercises only file and execution behavior.

Merge-risk options

Maintainer options:

  1. Preserve adapter verification through scoring (recommended)
    Keep the adapter context available while scoring, route state queries through its verifier, and add a stateful local-adapter regression test before merge.
Copy recommended automerge instruction 
@clawsweeper automerge

Special instructions:
Preserve the active adapter context through completion scoring, route state queries through verify_state_query, add coverage for a stateful local adapter, and do not alter the OpenClaw gateway path.

Technical review

Best possible solution:

Land the generic adapter lifecycle only after completion scoring uses the live adapter verifier for state queries and preserves the neutral capability-missing policy.

Do we have a high-confidence way to reproduce the issue?

Yes—source inspection provides a high-confidence path: run a local adapter that declares a state capability and verify that its successful state query is currently sent to the failing gateway shim instead.

Is this the best way to solve the issue?

No—the lifecycle should not substitute a gateway shim for the adapter-owned verification contract; retain the adapter/context through scoring and cover a stateful capability.

Full review comments:

  • [P1] Route state queries through the active adapter — clawbench/harness.py:644
    The local path tears down the adapter, then scores with _LocalVerificationClient; memory, session, cron, and gateway assertions therefore bypass verify_state_query and fail for any local adapter that supports them. Keep the adapter/context available through scoring and add stateful-adapter coverage. This is the prior blocker still present on the unchanged head.
    Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label justifications:

  • P2: This is a bounded benchmark-harness capability change with a concrete scoring defect but limited user-facing blast radius.
  • merge-risk: 🚨 session-state: Local adapter state assertions can be falsely failed because scoring does not invoke the adapter's state verifier.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🐚 platinum hermit and patch quality is 🦪 silver shellfish.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body records an after-fix local Hermes benchmark result; add redacted output for the stateful regression after the scoring repair.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body records an after-fix local Hermes benchmark result; add redacted output for the stateful regression after the scoring repair.

Evidence

Acceptance criteria:

  • [P1] python -m pytest tests/test_harness.py tests/test_hermes_adapter.py -q.
  • [P1] python -m pytest -q.

What I checked:

  • Current main does not implement the requested execution path: Current main explicitly rejects registered adapters that are not in the executable set, including Hermes, so this PR's central capability is not already implemented. (clawbench/harness.py:125, 884dd1bb5511)
  • Adapter teardown precedes local completion scoring: The PR tears down the adapter before calling completion scoring with _LocalVerificationClient, leaving no path for state assertions to invoke the active adapter verifier. (clawbench/harness.py:644, 67ff264005ab)
  • Completion scoring still uses gateway-based state checks: score_task_run passes its client to verify_completion, whose memory, session, cron, and gateway assertions call gateway methods rather than AgentAdapter.verify_state_query. (clawbench/scorer.py:99, 884dd1bb5511)
  • Added harness coverage is file-only: The new WritingAdapter declares only FILES and EXECUTION and its test uses a file task, so it cannot catch incorrect routing for a stateful adapter capability. (tests/test_harness.py:335, 67ff264005ab)
  • Adapter contract provenance: The canonical adapter layer introduced the lifecycle and adapter-owned state-query contract; the current PR should preserve that contract through scoring. (clawbench/adapters/base.py:20, 56531fbf432b)
  • After-fix local behavior evidence: The PR body reports a real Hermes rerun of t2-msg-summarize-thread improving from 0.63704 to 1.000; the prior review also recorded proof as sufficient.

Likely related people:

  • scoootscooob: Introduced the canonical adapter layer that defines adapter lifecycle and state-query verification responsibilities. (role: introduced adapter contract; confidence: high; commits: 56531fbf432b; files: clawbench/adapters/base.py, clawbench/adapters/hermes.py)
  • Vincent Koc: Authored the current scoring work adjacent to the gateway-based completion path this repair must extend safely. (role: recent scoring contributor; confidence: high; commits: fb486a1ed307; files: clawbench/scorer.py, clawbench/environment.py)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (37 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-03T04:09:35.877Z sha 67ff264 :: needs changes before merge. :: [P1] Pass adapter state verification into scoring
  • reviewed 2026-08-03T09:05:36.956Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the live adapter
  • reviewed 2026-08-03T20:14:07.544Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the live adapter verifier
  • reviewed 2026-08-04T15:02:17.830Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the live adapter verifier
  • reviewed 2026-08-04T18:03:35.221Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the live adapter verifier
  • reviewed 2026-08-04T21:04:17.796Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the live adapter verifier
  • reviewed 2026-08-04T23:02:55.304Z sha 67ff264 :: needs changes before merge. :: [P1] Route state assertions through the live adapter verifier
  • reviewed 2026-08-05T00:18:37.989Z sha 67ff264 :: needs changes before merge. :: [P1] Pass adapter verification into completion scoring

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Jul 1, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Jul 1, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. merge-risk: 🚨 session-state 🚨 Merging this PR could lose, corrupt, stale, or mis-associate session or agent state. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Jul 29, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. merge-risk: 🚨 session-state 🚨 Merging this PR could lose, corrupt, stale, or mis-associate session or agent state. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. merge-risk: 🚨 session-state 🚨 Merging this PR could lose, corrupt, stale, or mis-associate session or agent state. merge-risk: 🚨 other 🚨 Merging this PR has meaningful risk outside the owned taxonomy. labels Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

merge-risk: 🚨 session-state 🚨 Merging this PR could lose, corrupt, stale, or mis-associate session or agent state. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@afrog33k